This disclosure relates to an adaptive resource allocation solution for allocating resources in continuous stream data processing.
A modern stream data processing environment is quite complex. There are requirements to process, store, maintain and retrieve large volumes of mostly real-time data at high rates. The real-time data is referred to as continuous stream data. These requirements pose difficult design challenges for efficient stream processing systems.
Efficient stream processing systems have useful applications in large-scale reconnaissance, surveillance, environmental conditions monitoring, anomaly detection, and catastrophe control, for example. In these exemplary applications, the analysis usually must be done on-the-fly, with a low tolerance for delay.
There is therefore a need to find the most efficient way to allocate limited resources and to use such resources efficiently, so as to carry on the real-time processing.
In stream data processing environments, typically multiple data streams flow into the system and are processed to produce the desired valuable output. Examples of the processings that can be performed are matching, aggregation, summarization and the like. Typically, these processings can consist of several levels of interconnected tasks, such as represented in FIG. 1.
From FIG. 1 it is then seen that there is a requirement to determine the best resource allocation method from among the various processing tasks to optimize the overall performance. The presently known approaches only consider very simplistic models and provide ad hoc heuristics.